Device for wavelength-selective imaging

ABSTRACT

An imaging device captures both a visible light image and a diagnostic image, the diagnostic image corresponding to emissions from an imaging medium within the object. The visible light image (which may be color or grayscale) and the diagnostic image may be superimposed to display regions of diagnostic significance within a visible light image. A number of imaging media may be used according to an intended application for the imaging device, and an imaging medium may have wavelengths above, below, or within the visible light spectrum. The devices described herein may be advantageously packaged within a single integrated device or other solid state device, and/or employed in an integrated, single-camera medical imaging system, as well as many non-medical imaging systems that would benefit from simultaneous capture of visible-light wavelength images along with images at other wavelengths.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.10/517,280 filed Jun. 24, 2005, which is a national stage filing under35 U.S.C. 371 of International Appl. PCT/US03/16285 filed May 22, 2003,which claims priority to U.S. Appl. 60/382,524 filed May 22, 2002, eachof which are incorporated by reference herein. International Appl.PCT/US03/16285 was published under PCT Article 21(2) in English.

GOVERNMENT INTERESTS

The United States Government has certain rights in this inventionpursuant to Department of Energy Grant #DE-FG02-01ER63188.

BACKGROUND OF THE INVENTION

A number of medical imaging techniques have emerged for capturing stillor moving pictures using dyes that can be safely introduced into livingtissue. For example, fluorescent dyes may be adapted for sequestrationor preferential uptake at a location of medical interest, such as alesion. The location may then be exposed to a light source thatstimulates fluorescence of the dye to permit visualization that enhancesa feature of the location. Other emerging techniques employphosphorescent, chemoluminescent, or scintillant substances to generatephotons at one or more wavelengths suitable for imaging. Thesetechniques have proven useful for medical imaging and surveillance, withapplications including lesion imaging, calcium deposit imaging, andblood flow imaging.

Such imaging techniques have been enhanced with simultaneous capture andrendering of visible light images. This may, for example, provide anavigational tool at a surgical site, with the diagnostic image and thevisible light image superimposed for improved visualization.Charge-coupled devices (“CCDs”) provide one well-known system forconverting incident photons, or light, into a measurable electroniccharge. As a significant disadvantage, current CCD systems that combinevisible light and emission wavelength imaging typically employcommercially available components, and require at least two separatecameras: a first camera to capture the visible light image and a secondcamera for capturing the diagnostic emission wavelength which iscommonly, though by no means exclusively, in the near-infrared range. Atwo-camera system imposes the cost of an additional camera, as well asoptics for splitting the visual light wavelengths from the emissionwavelength and directing each to a separate transducer. There is alsoadditional software complexity and processing overhead in order tosynchronize and superimpose image data streams from the two cameras.

There remains a need for an integrated device that captures images fromvisible light wavelengths and diagnostic emission wavelengths.

SUMMARY OF THE INVENTION

An imaging device captures both a visible light image and a diagnosticimage, the diagnostic image corresponding to emissions from an imagingmedium within the object. The visible light image (which may be color orgrayscale) and the diagnostic image may be superimposed to displayregions of diagnostic significance within a visible light image. Anumber of imaging media may be used according to an intended applicationfor the imaging device, and an imaging medium may have wavelengthsabove, below, or within the visible light spectrum. The devicesdescribed herein may be advantageously packaged within a singleintegrated device or other solid state device, and/or employed in anintegrated, single-camera medical imaging system, as well as manynon-medical imaging systems that would benefit from simultaneous captureof visible-light wavelength images along with images at otherwavelengths.

In one aspect, the system includes a device that captures photonintensity from an illuminated object, the device being exposed to animage through a filter wheel including one or more filters thatselectively pass wavelengths of light to form a visible light image ofthe object and a filter that selectively passes wavelengths of light toform a diagnostic image of the object, the diagnostic imagecorresponding to emissions from an imaging medium within the object.

In another aspect, the system includes a plurality of devices thatcapture photon intensity from an illuminated object, the devices beingexposed to an image through a beam splitter and filters that selectivelypass incident photons along a number of paths according to wavelength,each one of the plurality of devices that capture photon intensity beingselectively exposed to an image including wavelengths passed along oneof the number of paths, at least one of the paths selectively passingwavelengths to form a diagnostic image of the object, the diagnosticimage corresponding to emissions from an imaging medium within theobject, and at least one of the paths selectively passing wavelengths toform a visible light image of the object.

In another aspect, the system includes a device that captures photonintensity from an illuminated object at a plurality of pixel locations,each one of the plurality of pixel locations covered by a filter, atleast one of the filters selectively passing wavelengths to form avisible light image of the object at a corresponding pixel location andat least one of the filters selectively passing wavelengths of light toform a diagnostic image of the object at a corresponding pixel location,the diagnostic image corresponding to emissions from an imaging mediumwithin the object.

In another aspect, the system includes a device that captures photonintensity from an illuminated object at a plurality of pixel locations,each one of the plurality of pixel locations including a plurality ofsuccessive diode junctions formed at the boundary of nested p-type andn-type semiconductor wells, each diode junction selectively detectingincident light over a range of wavelengths, at least one of the diodejunctions detecting wavelengths of a visible light image of the objectat that pixel location and at least one of the diode junctions detectingwavelengths of a diagnostic image of the object at that pixel location,the diagnostic image corresponding to emissions from an imaging mediumwithin the object.

The device that captures photon intensity may be a charge-coupleddevice. The device may consist of an integrated circuit. The imagingmedium may be a fluorescent dye, a phosphorescent substance, achemoluminscent substance, and/or a scintillant substance. The imagingmedium may be a substance introduced into the object. The imaging mediummay be a substance inherently present within the object. The object maybe an object within a surgical field. The visible light image may bemonochromatic. The visible light image may include red, blue and greenwavelengths of light. The visible light image may include cyan, magenta,and yellow wavelengths of light. The diagnostic image may include anear-infrared wavelength. The diagnostic image may include an infraredwavelength. The diagnostic image may include a plurality of diagnosticimages, each at a different range of wavelengths. The diagnostic imagemay be formed from one or more diagnostic wavelengths in the visiblelight range, the object being illuminated with a light source that isdepleted in the diagnostic wavelength range.

The visible light image and diagnostic image may be processed anddisplayed in a medical imaging system. The medical imaging system mayinclude a display for rendering a composite image including asuperposition of the visible light image and the diagnostic image. Themedical imaging system may include one or more inputs for controlling atleast one of a field of view of the object, a focus of the object, or azoom of the object. The medical imaging system may include a surgicaltool. The visible light image and diagnostic image may be processed anddisplayed in at least one of a machine vision system, an astronomysystem, a military system, a geology system, or an industrial system.

The system may be packaged in a camera. The camera may include a visiblelight image output, a diagnostic image output, and a combined imageoutput, the combined image output providing a superposition of thevisible light image and the diagnostic image. The system may capturemoving video, or the system may capture still images.

In another aspect, the system may include a solid state device thatcaptures a visible light image of an object under illumination indigital form and a diagnostic image of the object in digital form, thediagnostic image corresponding to an intensity of emission from animaging medium within the object.

In another aspect, the system may include a single camera that capturesa visible light image of an object under illumination and a diagnosticimage of the object, the diagnostic image corresponding to an intensityof emission from the object, the camera configured to provide a digitalversion of the visible light image and a digital version of thediagnostic image to an external display system.

In another aspect, a method may include the steps of illuminating anobject to provide an image; capturing an image of the object thatincludes a visible light image and a diagnostic image, the diagnosticimage corresponding to emissions from an imaging medium within theobject; and storing the image.

Capturing an image may include passing the image through a filter wheelthat exposes an image capture device to the image through a plurality offilters, at least one of the plurality of filters selectively passingwavelengths of light to form a visible light image of the object and atleast one of the plurality of filters selectively passing wavelengths oflight to form a diagnostic image of the object. Capturing an image mayinclude passing the image through a beam splitter and filters thatselectively pass incident photons along a number of paths according towavelength and exposing each one of a plurality of devices that capturephoton intensity to an image including wavelengths passed along one ofthe number of paths, at least one of the paths selectively passingwavelengths to form a diagnostic image of the object, and at least oneof the paths selectively passing wavelengths to form a visible lightimage of the object.

Capturing an image may include capturing the image at a plurality ofpixel locations, each one of the plurality of pixel locations covered bya filter, at least one of the filters selectively passing wavelengths toform a visible light image of the object at a corresponding pixellocation and at least one of the filters selectively passing wavelengthsof light to form a diagnostic image of the object at a correspondingpixel location. Capturing the image may include capturing the image at aplurality of pixel locations, each one of the plurality of pixellocations covered by a filter, at least one of the filters selectivelypassing wavelengths to form a visible light image of the object at acorresponding pixel location and at least one of the filters selectivelypassing wavelengths of light to form a diagnostic image of the object ata corresponding pixel location. Capturing the image may includecapturing the image at a plurality of pixel locations, each one of theplurality of pixel locations including a plurality of successive diodejunctions formed at the boundary of nested p-type and n-typesemiconductor wells, each diode junction selectively detecting incidentlight over a range of wavelengths, at least one of the diode junctionsdetecting wavelengths of a visible light image of the object and atleast one of the diode junctions detecting wavelengths of a diagnosticimage of the object at that pixel location.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be appreciated more fully from the following furtherdescription thereof, with reference to the accompanying drawings,wherein:

FIG. 1 is a block diagram of a prior art imaging system;

FIG. 2 is a block diagram of an imaging system with an integrated imagecapture device;

FIG. 3 depicts an embodiment of an image capture device;

FIG. 4 depicts an embodiment of an image capture device;

FIG. 5 is a side view of an image capture device on a single, integratedsemiconductor device;

FIG. 6 is a top view of an image capture device on a single, integratedsemiconductor device; and

FIG. 7 is a side view of an image capture device on a single, integratedsemiconductor device.

DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS OF THE INVENTION

To provide an overall understanding of the invention, certainillustrative embodiments will now be described, including an imagecapture device for simultaneously capturing visible-light andnear-infrared images. It will be understood that the methods and systemsdescribed herein can be suitably adapted to a range of medical imagingapplications where visible light tissue images may be usefully combinedwith diagnostic image information obtained from other specifiedwavelengths. For example, the systems may be applicable to a wide rangeof diagnostic or surgical applications where a target pathology, tissuetype, or cell may be labeled with a fluorescent dye or other fluorescentsubstance. More generally, the systems described herein may be adaptedto any imaging application where a visible light image may be usefullyenhanced with an image of one or more features that are functionallymarked to emit photons outside the visible light range by a dye or othermaterial that emits photons at a known wavelength, either inherently orin response to another known wavelength. The systems may also beemployed, for example, in a machine vision system, or in a variety ofother military, industrial, geological, astronomical or other imagingsystems. These and other applications of the systems described hereinare intended to fall within the scope of the invention.

FIG. 1 is a block diagram of a prior art imaging system. The imagingsystem 100 may include a light source 102 directed at a surgical field104, a near-infrared camera 106 receiving near-infrared light from thesurgical field 104 through a dichroic mirror 108, and a visible lightcamera 110 receiving visible light reflected from the dichroic mirror108. A processing and display system 112 receives data from the cameras106. A dye source (not shown) containing a dye may also be included forintroduction into an object in the surgical field 104, such as throughinjection into the bloodstream of a patient.

The light source 102 may include a visible light source and anexcitation light source that illuminate the surgical field 104.Preferably, the light source 102 is depleted in the wavelength regionwhere the dye or other emitting substance emits light, so that the lightsource 102 does not interfere with a diagnostic image obtained from thedye.

The near-infrared camera 106 captures a diagnostic image from anilluminated object within the surgical field at a wavelength or range ofwavelengths that pass through the dichroic mirror 108, while the visiblelight camera 110 captures a visible light image from the illuminatedobject within the surgical field. The visible light image may becaptured and rendered in a number of manners, such as “RGB”(red-green-blue), “CMY” (cyan-magenta-yellow), or monochromaticgrayscale. The diagnostic image corresponds to emissions from the dye orother imaging medium introduced to the object, such that the resultingimage is based, for example, upon the distribution of the dye in withinthe object in the surgical field.

It will be appreciated that visible light typically includes lightwavelengths from 400 nm to 700 nm, while near-infrared light may includeone or more wavelengths of about 810 nm, or more generally, wavelengthsfrom about 700 nm to about 1000 nm. In other variations, the emissionwavelength may be, for example, other near-infrared wavelengths, aninfrared wavelength, or a far red wavelength. More generally, theemission wavelength may be any wavelength of emission that can begenerated by an imaging medium introduced into an imaging subject andusefully captured for imaging with the devices described herein.

The imaging medium may include, for example, fluorescent dyes that emitin response to a stimulus wavelength, or a substance that emits photonsat a known wavelength without stimulus, such as phosphorescent,chemoluminescent, or scintillant substances. As one useful example,fluorescent substances, such as semiconductor nanocrystals (a.k.a.“quantum dots”) may be used to emit photons at one or more specificwavelengths, such as infrared wavelengths (e.g., 1320 nm). The imagingmedium may include substances inherently present in the object beingimaged, such as fluorescent or phosphorescent endogenous biologicalsubstances. In certain embodiments, the imaging medium may emit light ina range within the visible light spectrum. This may be usefully employedas a diagnostic imaging source, provided the light source 102 isadequately depleted, such as through filtering, in a correspondingwavelength range so that the light source 102 does not produce reflectedlight within the diagnostic image wavelength range. The term “imagingmedium” as used herein, refers to any of the imaging media describedabove, or any other medium capable of emitting photons useful inlocating regions of functional or diagnostic significance. A “diagnosticimage” as that term is used herein, refers to any image formed bydetecting emissions from the imaging media described above.

Once captured, each diagnostic image may be shifted to a suitablevisible light wavelength for purposes of display. In one embodiment,this pseudo-coloring employs a color specifically selected to provide asubstantial color contrast with the object of the visible light image. Acolor may be selected in advance, such as bright lime green for adiagnostic image over living tissue, or the color may be determinedautomatically by an algorithm designed to determine average backgroundcolor and choose a suitable contrasting color. The visible light anddiagnostic images may be combined by the image processing and displaysystem 112 and presented on a display where they may be used, forexample, by a surgeon conducting a surgical procedure. The processingand display system 112 may include any suitable hardware and software tocombine and render the diagnostic and visible light images in any manneruseful for a user of the system 100, such as a composite image formedfrom a superposition of the diagnostic and visible light images, orside-by-side rendering of the diagnostic and visible light images.

The processing and display system 112 may be part of a medical imagingsystem that also includes, for example, inputs to control visualnavigation of the surgical field, such as field of view (e.g., X and Ypanning), zoom, and focus, or inputs for controlling a surgical toolassociated with the system 100. Similar controls may be provided for thenon-medical applications noted above, with certain adaptations asappropriate, such as azimuth and elevation in place of X/Y panning forastronomical applications.

It will be appreciated that certain of the terms and concepts introducedabove are applicable to some or all of the embodiments described below,such as the terms diagnostic image and imaging medium, as well as thenature of the processing and display system, except as specificallynoted below. It will also be appreciated that adaptations may be made.For example, where a single, integrated camera is provided for capturingboth a visible light image and a diagnostic image, some of the imageprocessing for pseudo-coloring the diagnostic image and superimposingthe diagnostic image onto the visible light image may be provided by thecamera, with an output of the processed image provided in any suitableformat to a computer, display, or medical imaging system.

FIG. 2 is a block diagram of an imaging system with an integrated imagecapture device. The imaging system 200 may include a light source 202directed at a surgical field 204, an image capture device 206 receivinglight from the surgical field 204, and a processing and display system208. A dye source (not shown) containing in imaging medium such as a dyemay also be included for introduction into an object in the surgicalfield 204, such as through injection into the bloodstream of a patient.The imaging system 200 may be in many respects like the imaging system100 described above with reference to FIG. 1. It will readily beappreciated that the imaging system 200 differs in at least onerespect—the use of a single image capture system 206.

The system 200 advantageously incorporates visible light and diagnosticwavelength imaging into a single device, the image capture system 206.This removes the need for additional external hardware, such as thedichroic mirror 108 of FIG. 1, or additional hardware and/or software inthe processing and display system 208 to perform additional processingfor images from two separate image capture devices, which processing mayrange from image registration to matching of frame rates, image sizes,and other features of images from disparate cameras. The image capturesystem 206 may be packaged as a single solid state device suitable forintegration into a larger system, or as a camera with inputs for remoteoperation and/or outputs including a visible light output, a diagnosticimage output, and a combined output that superimposes the diagnostic andvisible light images.

In certain embodiments, the image capture system 206 may provide forcapture of two or more wavelengths of diagnostic significance throughadaptations of the systems described below. Thus two or more diagnosticimages may be displayed, and/or superimposed on a visible light image inorder to simultaneously visualize two or more features orcharacteristics of interest. The image capture system 206 may providestill images, or may provide moving images, such as in a real-timedisplay of a surgical field.

A number of technologies may be suitable adapted to the image capturesystem 206. Charge-Coupled Devices (“CCDs”), for example, are known foruse in capturing digital images. These devices may be fabricated onsilicon substrates and packaged as chips, employing various CCDtechnologies. For example, full-frame-transfer (“FF”) and frame-transfer(“FT”) devices employ MOS photocapacitors as detectors, while interlinetransfer (“IL”) devices use photodiodes and photocapacitors for eachdetector. These architectures are among the more commonly employedarchitectures in current CCD cameras. Each CCD technology has its ownadvantages and disadvantages, resulting in trade-offs between, forexample, cost, design complexity, and performance. These technologiesare generally adaptable to the systems described herein, and carry withthem the corresponding design trade-offs, as will be appreciated bythose of skill in the art.

Other image-sensing architectures using charge-coupled devices areknown, and may be usefully employed with the systems described herein,including frame-interline transfer devices, accordion devices, chargeinjection devices, and MOS X, Y addressable devices. All such devicesare intended to fall within the meaning of “charge-coupled device” asthat term is used herein. While all of these devices are useful forconverting incident photons into measurable electronic charges, they areinherently monochromatic in nature. As such, color-imaging applicationshave been devised for these CCDs that selectively image differentwavelengths. These techniques for wavelength selection may be adapted tothe present system as described in greater detail below.

FIG. 3 depicts an embodiment of an image capture device. The device 300applies an adaptation of mechanical color wheels used for someconventional red-green-blue (“RGB”) imaging systems. A image of anilluminated object may be focused through a lens and captured in foursuccessive exposures, each synchronized with a filter wheel 302 havingdesired optical characteristics. In the depicted embodiment, thisincludes a red filter 304 that selectively passes red light, a greenfilter 306 that selectively passes green light, a blue filter 308 thatselectively passes blue light, and a near-infrared filter 310 thatselectively passes near-infrared light. The CCD 312 is exposed to theimage through the red, green, and blue filters 304, 306, 308collectively to capture a visible light image, and exposed to the imagethrough the near-infrared filter 310 that selectively passesnear-infrared emissions to capture a diagnostic image of interest, suchas emission from a fluorescent dye.

Each exposure of the CCD 312 is sequentially read into data storage (notshown) where it can be reconstructed into a complete image. It will beappreciated that a number of other wavelengths may be selectively passedby the fourth filter to obtain a diagnostic image of the object,including infrared wavelengths or other wavelengths of interest, asgenerally described above. It will further be appreciated thatadditional filters may be added to the color wheel so that two or moreemission wavelengths may be captured within the same image. Thus a colorwheel with five or more filters is contemplated by the systems describedherein.

In another aspect, a method according to the above system may includecapturing an image that passes through the filter wheel 302 on the CCD312 or other image capture device to obtain a visible light image and adiagnostic image.

FIG. 4 depicts an embodiment of an image capture device. As shown in thefigure, a multi-chip device 400 may employ optics to split an image intoseparate image planes. A focused image is provided to the device 400,such as by passing an image of an illuminated object through a lens. Aplurality of CCDs 402 or other image capture devices for measuringphoton intensity are exposed to the image through a beam splitter 404,with a CCD 402 placed in each image plane exiting the beam splitter 404.A filter may also be provided for each CCD 402 to selectively expose theCCD 402 to a range of wavelengths, so that the image is selectivelypassed along a number of paths according to wavelengths. It will beappreciated that other similar approaches may be used to apply differingwavelengths to a collection of CCDs 402 in a multi-chip CCD system, suchas a prism or a wavelength separating optical device or devices. In suchsystems, the CCDs 402 may be operated synchronously to capture differentincident wavelengths at or near the same point in time.

In FIG. 4, the beam-splitter 404 provides four different CCDs 402, afirst CCD with a filter that passes red light, a second CCD with afilter that passes green light, a third CCD with a filter that passesblue light, and a fourth CCD with a filter that passes near-infraredlight. The first three CCDs produce a visible light image, while thefourth CCD produces a diagnostic image according to an imaging mediumintroduced into the object. However, It will be appreciated that otherwavelengths may be passed by the fourth filter, including infraredwavelengths or other wavelengths of interest. It will further beappreciated that additional light paths may be provided by the beamsplitter with additional filtered CCDs for each path, so that two ormore emission wavelengths may be captured within the same image. Thus asystem with five or more CCDs is contemplated by the systems describedherein.

In another aspect, a method according to the above system may includecapturing an image that passes through the beam splitter 404 and filtersthat selectively pass incident photons along a number of paths accordingto wavelength, with each CCD (or other image capture device) capturingeither a visible light image or a diagnostic image of the illuminatedobject.

FIG. 5 is a side view of an image capture device on a single, integratedsemiconductor device. The figure shows a CCD array 500 with wavelengthselection using an integral filter array. The CCD array 500 may includelenses 502, a filter array 504, gates 506, photodiodes 508, a substrate510, vertical charge-coupled devices (“VCCDs”) 512, and an insulationlayer 514.

In the CCD array 500, the photodiodes 508 serve to detect the intensityof incident photons at pixel locations within a focused image, while thefilter array 504 with appropriate characteristics are arranged over thephotodiodes 508 such that different photodiodes are exposed to differentwavelengths of incident light. The filter array 504 may include, forexample red filters that selectively pass red wavelengths (labeled “R”),green filters that selectively pass green wavelengths (labeled “G”),blue filters that selectively pass blue wavelengths (labeled “B”), andnear-infrared filters that selectively pass near-infrared wavelengths(labeled “I”). The VCCDs 512 may be formed vertically between thephotodiodes 508 for transferring signals produced by photoelectricconversion in the photodiodes 508. The insulation layer 514 may beformed over the entire surface of the semiconductor substrate 510(including the photodiodes 508 and the VCCD 512), and the plurality ofgates 506 may be formed on the insulation layer 514 above each VCCD 512for controlling the transfer of the photodiode signals. A metalshielding layer for shielding light may be deposited over the gates 512,except for the light-receiving regions of the photodiodes 508. A flatinsulation film may then be deposited over the entire surface of thesemiconductor substrate including the metal shielding layer.

The filter array 504, which passes either red (“R”), green (“G”), blue(“B”) (collectively for forming a visible light image), or near-infrared(“NI”) wavelengths (for a diagnostic image) may then be formed over eachphotodiode 508 corresponding to a pixel to be imaged from an illuminatedobject. A top coating layer may be deposited on the filter layer 504.Finally, a lens 502 may be formed on the top coating layer forconcentrating photons on each photodiode.

The filter array 504 separates the spectrum of incident light toselectively pass only the light of a predetermined wavelength, or rangeof wavelengths, to reach each of the photodiodes. The metal shieldinglayer restricts incident light to the photodiodes 508. The incidentlight is converted into an electric signal in the photodiodes 508, andtransferred out to a processor under control of the gates 506.

The filter array 504 may be dyed or otherwise masked or processed sothat each photodiode 508 is exposed to a specific wavelength or range ofwavelengths. It will be appreciated that the device of FIG. 5 is anexample only, and that a number of different CCD topologies may be usedwith an integral filter array 504, and may be suitably adapted to thesystems described herein. It should also be appreciated that otherphotoactive substances may be included in place of, or in addition to,the filters in the filter array 504, in order to enhance response atcertain wavelengths, or to affect a shift in wavelength to a moresuitable frequency for measurement by the photodiodes. All suchvariations are intended to fall within the scope of this description.

FIG. 6 is a top view of an image capture device on a single, integratedsemiconductor device. The figure depicts one possible arrangement offilters 602 for use with the systems described herein. In this integralfilter array 602, four filters are arranged to expose photodiodes to red(labeled “R”), green (labeled “G”), blue (labeled “B”), andnear-infrared (labeled “I”) wavelengths. Each two-by-two group ofphotodiodes may form a pixel, with four wavelength measurements beingdetected for that pixel at different photodiodes.

It will be appreciated that a number of other wavelengths may be passedby the fourth filter (“I”), including infrared wavelengths or otherwavelengths of interest. It will further be appreciated that additionalfilters may be disposed upon the CCD, with suitable adjustments to thearrangement of filters, so that two or more emission wavelengths may becaptured within the same image. Thus a system with an integral filterfor five or more wavelengths is contemplated by the systems describedabove.

In another aspect, a method according to the above system may includecapturing an image that passes through a filter array that selectivelypasses wavelengths of either a visible light image or a diagnostic imageto a pixel location in a charge coupled device.

FIG. 7 is a side view of an image capture device on a single, integratedsemiconductor device. As shown in the figure, the device 700 may includea number of nested p-type and n-type wells, with a p-n diode junctionformed at each well boundary that is sensitive to incident photons of aparticular wavelength range. By measuring current across these p-njunctions while the device 700 is exposed to light, photon intensityover a number of contiguous wavelengths may be detected at the samelocation at the same time.

More specifically, the device 700 includes an n-type substrate 702, ap-well 704 within the n-type substrate, an n-well 706 within the p-well704, a p-well 708 within the n-well 706, and an n-drain 710 within thep-well 708. A first detector 712 measures photocurrent across a firstp-n junction 714 and is generally sensitive to blue wavelengths. Asecond detector 716 measures photocurrent across a second p-n junction718, and is generally sensitive to green wavelengths. A third detector720 measures photocurrent across a third p-n junction 722, and isgenerally sensitive to red wavelengths. A fourth detector 724 measuresphotocurrent across a fourth p-n junction 726, and is generallysensitive to an emission wavelength from an imaging medium within anobject.

A similar, triple-well structure is described, for example, in U.S. Pat.No. 5,965,875 to Merrill. In general, such a device operates on theprinciple that photons of longer wavelengths will penetrate more deeplyinto silicon before absorption. By alternately doping wells for p-typeor n-type conductivity, a number of successive photodiodes are createdat the p-n junctions of successive layers, each being sensitive toprogressively longer wavelengths of photon emissions. Using well-knownactive pixel technology to sense photocurrents from these diodes (asshown by circuits labeled “iB”, “iG”, “iR”, and “iNI”), each activepixel region senses photocharge by integrating the photocurrent on thecapacitance of a photodiode and the associated circuit node, and thenbuffering the resulting voltage through a readout amplifier. A shallow,n-type, lightly-doped drain above the first p-type well may be employedto maximize blue response in the first photodiode.

The above quadruple-well system may be advantageously adapted to imagingsystems where an emission wavelength is adjacent to, or nearly adjacentto, the visible light spectrum. The near-infrared spectrum, for example,is adjacent to the red wavelength spectrum, and may be measured with thedevice described above.

In another aspect, a method according to the above system may includecapturing photon intensity from an illuminated object at a pixellocation at a number of different wavelengths by measuring photocurrentat a plurality of successive diode junctions formed at the boundary ofsuccessively nested p-type and n-type semiconductor wells.

The near-infrared spectrum lies in a range that is particularly usefulfor certain medical imaging applications, due to the low absorption andautofluorescence of living-tissue components in this range. Within arange of 700 nm to 900 nm, the absorbances of hemoglobin, lipids, andwater reach a cumulative minimum. This so-called “near-infrared window”provides a useful spectrum for excitation and emission wavelengths inliving-tissue imaging applications, and a number of fluorescent dyesusing these wavelengths have been developed for medical imagingapplications. Thus the quadruple-well device described above not onlyemploys a convenient range of wavelengths adjacent to visible light, itaccommodates a number of dyes that are known to be safe and effectivefor tissue imaging, such as the IR-786 or the carboxylic acid form ofIRDye-78, available from LI-COR, Inc.

It will be appreciated that each of the systems described above presentstrade-offs in terms of cost, speed, image quality, and processingcomplexity. For example, the single-CCD filter wheel may introducesignificant time delays between different wavelength images, and may notperform well in high-speed imaging applications. By contrast, themulti-chip approach requires more CCD elements and additional processingin order to maintain registration and calibration between separatelyobtained images, all of which may significantly increase costs. As such,different applications of the systems described herein may havedifferent preferred embodiments.

The systems described above have numerous surgical applications whenused in conjunction with fluorescent dyes. For example, the system maybe deployed as an aid to cardiac surgery, where it may be usedintraoperatively for direct visualization of cardiac blood flow, fordirect visualization of myocardium at risk for infarction, and forimage-guided placement of gene therapy and other medicinals to areas ofinterest. The system may be deployed as an aid to oncological surgery,where it may be used for direct visualization of tumor cells in asurgical field or for image-guided placement of gene therapy and othermedicinals to an area of interest. The system may be deployed as an aidto general surgery for direct visualization of any function amenable toimaging with fluorescent dyes, including blood flow and tissueviability. In dermatology, the system may be used for sensitivedetection of malignant cells or other skin conditions, and fornon-surgical diagnosis of dermatological diseases using near-infraredligands and/or antibodies. More generally, the CCD systems describedherein may be used as imaging hardware in conjunction with open-surgicalapplications, and may also be integrated into a laparoscope, anendoscope, or any other medical device that employs' an imaging system.The systems may have further application in other non-medical imagingsystems that combine visible light and non-visible light imaging.

In various embodiments, the system described herein includes awavelength-selective solid state device, such as a CCD or othersemiconductor device, a semiconductor chip that includes the solid statedevice, a camera employing the solid state device, an imaging systememploying the solid state device, and methods of imaging that employ thesolid state device. A camera using the imaging devices described abovemay include a lens, user inputs or a wired or wireless remote controlinput, the imaging device, processing to filter, store and otherwisemanage captured images including functions such as superposition ofdiagnostic and visible light images and pseudo-coloring of thediagnostic image, and one or more outputs for providing the images to aremote device or system.

It will be appreciated that certain imaging technologies are moresuitable to capturing certain wavelengths, including technologies suchas gas photodiode or microplasma photodetectors, and certain substancesor combinations of substances, such as Indium, Gallium, or Germanium mayprovide enhanced responsiveness over certain wavelength ranges. Some ofthese are consistent with CMOS manufacturing and may be realizeddirectly on a wafer with visible-light-imaging circuitry and otherprocessing circuitry, or manufactured with micro-electro-mechanicalsystems technology and packaged within the same chip as relatedcircuitry, or the solid state system may be provided as a chipset thatis assembled and provided on a suitable circuit board. All suchtechnologies as may be useful for visible light imaging and/ordiagnostic imaging over the wavelengths described above may be used forthe solid state devices described above, and are intended to fall withinthe scope of the invention.

As a significant advantage, cameras using the devices described hereinmay receive an image through a single lens, and provide both visiblelight and diagnostic images on a single output. As another significantadvantage, visible light and diagnostic images may be obtained from asingle, solid-state device, reducing the requirement for moving parts,additional lenses and expensive optics, and post-processing associatedwith combining images from different sources.

While medical imaging applications have been described, it will beappreciated that the principles of the systems described above may bereadily adapted to other applications, such as machine vision or avariety of other military, industrial, geological, astronomical or otherimaging systems. For example, a machine vision system may employ afluorescent dye that selectively adheres to surfaces of a certaintexture, or aggregates in undesirable surface defects. A diagnosticimage of the dye may assist in identifying and/or repairing theselocations.

Thus, while the invention has been disclosed in connection with thepreferred embodiments shown and described in detail, variousmodifications and improvements thereon will become readily apparent tothose skilled in the art. It should be understood that all mattercontained in the above description or shown in the accompanying drawingsshall be interpreted as illustrative, and not in a limiting sense, andthat the invention should be interpreted in the broadest sense allowableby law.

1. A system comprising; a plurality of solid state devices that capturephoton intensity from an illuminated object, the solid state devicesbeing exposed to an image of the illuminated object through a beamsplitter and filters that selectively pass incident photons along anumber of paths according to wavelength, each one of the solid statedevices being selectively exposed to a portion of the image includingwavelengths passed along one of the number of paths, at least one of thepaths selectively passing infrared wavelengths to form a diagnosticimage of the illuminated object at one of the solid state devices thatmonochromatically represents an intensity of infrared wavelengths fromthe illuminated object corresponding to emissions from an imaging mediumwithin the illuminated object, and at least one other one of the pathsselectively passing wavelengths to another one of the solid statedevices to form a visible light image of the illuminated object; animage processing system configured to pseudocolor the diagnostic imageto provide a pseudocolored diagnostic image, and configured tosuperimpose the pseudocolored diagnostic image onto the visible lightimage to provide a processed image; and a camera containing a lens, theplurality of solid state devices, and the image processing system, thecamera further including one or more inputs for remote operation of thecamera and plurality of outputs for an external display system, theplurality of outputs including a visible light output for the visiblelight image, a diagnostic image output for the diagnostic image, acombined output for the processed image.
 2. The system of claim 1wherein the imaging medium is at least one of a fluorescent dye, aphosphorescent substance, a chemoluminscent substance, or a scintillantsubstance.
 3. The system of claim 1 wherein the imaging medium is asubstance introduced into the illuminated object.
 4. The system of claim1 wherein the imaging medium is a substance inherently present withinthe illuminated object.
 5. The system of claim 1 wherein the illuminatedobject is an object within a surgical field.
 6. The system of claim 1wherein the visible light image includes red, blue and green wavelengthsof light.
 7. The system of claim 1 wherein the visible light imageincludes cyan, magenta, and yellow wavelengths of light.
 8. The systemof claim 1 wherein the diagnostic image includes a near-infraredwavelength.
 9. The system of claim 1 wherein the diagnostic imageincludes a plurality of diagnostic images, each at a different range ofwavelengths.
 10. The system of claim 1 wherein the diagnostic image isformed from one or more diagnostic wavelengths in the visible lightrange, the illuminated object being illuminated with a light source thatis depleted in the diagnostic wavelength range.
 11. The system of claim1 wherein the visible light image and diagnostic image are processed anddisplayed in a medical imaging system.
 12. The system of claim 11,further comprising a display adapted to receive and render the processedimage from the camera.
 13. The system of claim 11 wherein the one ormore inputs for the camera control at least one of a field of view ofthe illuminated object, a focus of the illuminated object, or a zoom ofthe illuminated object.
 14. The system of claim 1 wherein the visiblelight image and diagnostic image are processed and displayed in at leastone of a machine vision system, an astronomy system, a military system,a geology system, or an industrial system.
 15. The system of claim 1wherein the camera is a video camera that capture moving video.
 16. Thesystem of claim 1 wherein camera captures still images.
 17. The systemof claim 1 further comprising a visible light source positioned toilluminate the illuminated object.
 18. The system of claim 17 whereinthe visible light source is depleted in a region corresponding to thediagnostic image.
 19. The system of claim 1 further comprising anexcitation light source having an emission wavelength selected to excitethe imaging medium and positioned to illuminate the illuminated object.20. The system of claim 1 wherein the plurality of solid state devicesinclude a plurality of charge-coupled devices.